Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus includes: a normal interpolated image generation unit to generate an image that is interpolated between a plurality of original images reproduced along time series, the image being a normal interpolated image, based on each of the plurality of original images; a high-frequency area extraction unit to extract a high-frequency area having a spatial frequency higher than a predetermined value in each of the plurality of original images; a high-frequency area interpolated image generation unit to generate an image that is interpolated between the plurality of original images, the image being a high-frequency area interpolated image, based on a change in position of the high-frequency area along with an elapse of time on the time series and on each of the plurality of original images; and a combination unit to execute combining processing to combine the normal interpolated image and the high-frequency area interpolated image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2012-277561 filed Dec. 20, 2012, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an image processing apparatus, animage processing method, and a program causing a computer to execute theimage processing method. Specifically, the present disclosure relates toan image processing apparatus and an image processing method with whichthe number of frames per unit time in a moving image is changed and to aprogram causing a computer to execute the image processing method.

In image processing apparatuses in related art such as televisionreceivers, a frame rate conversion to change the number of frames perunit time, i.e., a frame rate, in a moving image is performed in somecases. For example, in order to smoothen the motion of an object in amoving image, a frame rate conversion to increase a frame rate, which iscalled an up-conversion, is performed.

In such an up-conversion, a frame interpolation method in which aninterpolated frame is generated from original frames and insertedbetween the original frames is widely used. In the frame interpolationmethod, motion compensation processing to detect the motion of an objectin the original frame and generate an interpolated frame based on themotion is performed. This allows a natural interpolation to be performedin a moving image including a moving object.

In performing the motion compensation processing, a method of detectingthe motion of an object on a block-to-block basis or a pixel-to-pixelbasis is widely used, but the method may cause a possibility that themotion detection fails in some blocks or pixels. If the object whosemotion has failed to be detected is a high-frequency component, e.g., acharacter telop, a defect easily occurs in an image with a part at whichthe detection has failed. For example, since an object including ahigh-frequency component has a clear boundary with the background, blursoccur at the part at which the detection has failed. On the other hand,since an object including a low-frequency component has a blurredboundary, blurs are not liable to occur if the motion detection fails.

To suppress the defect involving the failure of the motion detection,the following image processing apparatus is provided. The imageprocessing apparatus detects an area including a high-frequencycomponent with motion to be an area for character telop and changes aninterpolation method for such an area (see, for example, Japanese PatentApplication Laid-open No. 2009-296284). The image processing apparatusdetects motion of the entire area in the area of the character telop anddetects motion on a block-to-block basis or a pixel-to-pixel basis inthe other areas. Subsequently, along the direction of the detectedmotion, the entire area of the character telop in the original frame isdisplaced, and in areas other than the character telop, the originalframe is displaced for each block or pixel, to generate an interpolatedframe. The entire area of the character telop is displaced, so that noblurs occur in the area.

SUMMARY

In the above-mentioned technique in related art, however, the imagequality of the moving image after the frame rate conversion maydeteriorate. Specifically, the image processing apparatus performsdifferent types of processing on the area of the character telop and onthe other areas, and hence there is a possibility that the boundariesbetween those areas look unnatural.

This leads to a possibility that the image quality deteriorates.

The present disclosure has been made in view of the circumstances asdescribed above, and it is desirable to improve the image quality of amoving image whose frame rate is converted.

According to an embodiment of the present disclosure, there are providedan image processing apparatus, an image processing method therefor, anda program causing a computer to execute the image processing method. Theimage processing apparatus includes a normal interpolated imagegeneration unit, a high-frequency area extraction unit, a high-frequencyarea interpolated image generation unit, and a combination unit. Thenormal interpolated image generation unit is configured to generate animage that is interpolated between a plurality of original imagesreproduced along time series, the image being a normal interpolatedimage, based on each of the plurality of original images. Thehigh-frequency area extraction unit is configured to extract ahigh-frequency area having a spatial frequency higher than apredetermined value in each of the plurality of original images. Thehigh-frequency area interpolated image generation unit is configured togenerate an image that is interpolated between the plurality of originalimages, the image being a high-frequency area interpolated image, basedon a change in position of the high-frequency area along with an elapseof time on the time series and on each of the plurality of originalimages. The combination unit is configured to execute combiningprocessing to combine the normal interpolated image and thehigh-frequency area interpolated image. This provides effects ofgenerating the normal interpolated image based on each of the pluralityof original images, generating the high-frequency area interpolatedimage based on the change in position of the high-frequency area alongwith an elapse of time on the time series and on each of the pluralityof original images, and combining the normal interpolated image and thehigh-frequency area interpolated image.

Further, in the embodiment of the present disclosure, the imageprocessing apparatus may further include a vector detection unitconfigured to detect a vector indicating a direction and a distance inwhich a position of the high-frequency area changes within apredetermined period of time on the time series, and the high-frequencyarea interpolated image generation unit may be configured to generatethe high-frequency area interpolated image based on the vector and eachof the plurality of original images. This provides an effect ofgenerating the high-frequency area interpolated image based on thevector indicating the direction and the distance in which the positionof the high-frequency area changes and on each of the plurality oforiginal images.

Further, in the embodiment of the present disclosure, the imageprocessing apparatus may further include a combination ratiodetermination unit configured to determine a ratio at which the normalinterpolated image and the high-frequency area interpolated image arecombined, for each pixel in accordance with each pixel value of pixelsin the high-frequency area, and the combination unit may be configuredto execute the combining processing according to the ratio. Thisprovides an effect of executing the combining processing according tothe ratio determined for each pixel in accordance with each pixel valueof the pixels in the high-frequency area.

Further, in the embodiment of the present disclosure, the high-frequencyarea extraction unit may include a filter unit, a difference detectionunit, and an increase unit. The filter unit is configured to extract thehigh-frequency area in each of the plurality of original images. Thedifference detection unit is configured to detect, for each pixel, adifference in pixel value between two adjacent original images of theplurality of original images. The increase unit is configured toincrease each pixel value of the pixels in the high-frequency area ineach of the two adjacent original images in accordance with thedifference, and supply the increased pixel value to the vector detectionunit and the combination ratio determination unit. This provides aneffect of increasing each pixel value of the pixels in thehigh-frequency area in accordance with the difference.

Further, in the embodiment of the present disclosure, the high-frequencyarea extraction unit may include a first difference coring processingunit configured to execute first difference coring processing to removea difference smaller than a first difference threshold from the detecteddifferences, and the increase unit may be configured to increase eachpixel value of the pixels in the high-frequency area in accordance withthe difference on which the first difference coring processing isperformed, and supply the increased pixel value to the vector detectionunit. This provides an effect of increasing each pixel value of thepixels in the high-frequency area in accordance with the difference onwhich the first difference coring processing is performed.

Further, in the embodiment of the present disclosure, the high-frequencyarea extraction unit may include a second difference coring processingunit configured to execute second difference coring processing to removea difference smaller than a second difference threshold from thedetected differences, and the increase unit may be configured toincrease each pixel value of the pixels in the high-frequency area inaccordance with the difference on which the second difference coringprocessing is performed, and supply the increased pixel value to thecombination ratio determination unit. This provides an effect ofincreasing each pixel value of the pixels in the high-frequency area inaccordance with the difference on which the second difference coringprocessing is performed.

Further, in the embodiment of the present disclosure, the high-frequencyarea extraction unit may include a first pixel coring processing unitconfigured to execute first pixel coring processing to remove a pixelwith a pixel value smaller than a first pixel value threshold in theextracted high-frequency area, and the increase unit may be configuredto increase, in accordance with the difference, each pixel value of thepixels in the high-frequency area on which the first pixel coringprocessing is performed, and supply the increased pixel value to thevector detection unit. This provides an effect of increasing, inaccordance with the difference, each pixel value of the pixels in thehigh-frequency area on which the first pixel coring processing isperformed.

Further, in the embodiment of the present disclosure, the high-frequencyarea extraction unit may include a second pixel coring processing unitconfigured to execute second pixel coring processing to remove a pixelwith a pixel value smaller than a second pixel value threshold in theextracted high-frequency area, and the increase unit may be configuredto increase, in accordance with the difference, each pixel value of thepixels in the high-frequency area on which the second pixel coringprocessing is performed, and supply the increased pixel value to thecombination ratio determination unit. This provides an effect ofincreasing, in accordance with the difference, each pixel value of thepixels in the high-frequency area on which the second pixel coringprocessing is performed.

Further, in the embodiment of the present disclosure, each of theplurality of original images may include a plurality of blocks eachhaving a predetermined shape, and the normal interpolated imagegeneration unit may be configured to generate the normal interpolatedimage based on a change in position of each of the plurality of blocksalong with an elapse of time on the time series and on each of theplurality of original images. This provides an effect of generating anormal interpolated image based on the change in position of each of theplurality of blocks along with an elapse of time on the time series andon each of the plurality of original images.

Further, in the embodiment of the present disclosure, the imageprocessing apparatus may further include a selection unit configured toselect the combined image by the combination unit and the plurality oforiginal images in order of the time series and output the selectedimage. This provides an effect of selecting the combined image and theplurality of original images in order of the time series and outputtingthe selected image.

According to the present disclosure, it is possible to produce anexcellent effect of improving the image quality of a moving image whoseframe rate is converted.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a configuration example of aninformation processing system in a first embodiment;

FIG. 2 is a block diagram showing a configuration example of an imageprocessing apparatus in the first embodiment;

FIG. 3 is a block diagram showing a configuration example of aninterpolated image generation unit in the first embodiment;

FIGS. 4A, 4B, and 4C are each a diagram for describing a method ofdetecting a local motion vector in the first embodiment;

FIG. 5 is a diagram showing an example of the local motion vector in thefirst embodiment;

FIG. 6 is a block diagram showing a configuration example of a normalinterpolated image generation unit in the first embodiment;

FIGS. 7A, 7B, and 7C are each a diagram for describing a method ofgenerating a normal interpolated image in the first embodiment;

FIG. 8 is a block diagram showing a configuration example of ahigh-frequency area extraction unit in the first embodiment;

FIG. 9 is a block diagram showing a configuration example of adifference detection unit in the first embodiment;

FIG. 10 is a block diagram showing a configuration example of ahigh-frequency area interpolated image generation unit in the firstembodiment;

FIG. 11 is a block diagram showing a configuration example of acombination ratio determination unit in the first embodiment;

FIG. 12 is a block diagram showing a configuration example of acombination unit in the first embodiment;

FIG. 13 is a diagram showing an example of the normal interpolated imagein the first embodiment;

FIG. 14 is a diagram showing an example of a high-frequency areainterpolated image in the first embodiment;

FIG. 15 is a diagram showing an example of combination ratio data in thefirst embodiment;

FIG. 16 is a diagram showing an example of a combined interpolated imagein the first embodiment;

FIG. 17 is a flowchart showing an example of an operation of the imageprocessing apparatus in the first embodiment;

FIG. 18 is a flowchart showing an example of interpolated imagegeneration processing in the first embodiment;

FIG. 19 is a block diagram showing a configuration example of aninterpolated image generation unit in a modified example of the firstembodiment;

FIG. 20 is a block diagram showing a configuration example of a normalinterpolated image generation unit in the modified example of the firstembodiment;

FIG. 21 is a diagram showing an example of a normal interpolated imagein the modified example of the first embodiment;

FIG. 22 is a block diagram showing a configuration example of aninterpolated image generation unit in a second embodiment;

FIG. 23 is a block diagram showing a configuration example of ahigh-frequency area extraction unit in the second embodiment; and

FIG. 24 is a block diagram showing a configuration example of ahigh-frequency area extraction unit in a modified example of the secondembodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, modes for carrying out the present disclosure (hereinafter,referred to as embodiments) will be described. Description will be givenin the following order.

1. First Embodiment (Example of Combining Normal Interpolated Image andHigh-Frequency Area Interpolated Image)

2. Second Embodiment (Example of Combining Normal Interpolated Image andHigh-Frequency Area Interpolated Image Generated from DifferenceObtained After Coring Processing)

1. First Embodiment Configuration Example of Information ProcessingSystem

FIG. 1 is a block diagram showing a configuration example of aninformation processing system in a first embodiment. The informationprocessing system includes a moving image supply apparatus 100, an imageprocessing apparatus 200, and a display apparatus 400.

The moving image supply apparatus 100 supplies an input signal, whichcontains a moving image, to the image processing apparatus 200 via asignal line 109. The moving image contains a plurality of frames(images) to be reproduced in a time-series order. Hereinafter, each ofthe images in the input signal is referred to as an “original image”.For example, the moving image supply apparatus 100 receives a broadcastwave providing a moving image and acquires the moving image from thebroadcast wave. When the frame rate of the moving image is representedby hertz (Hz), the frame rate of the moving image in the input signal is60 hertz, for example.

Note that the moving image supply apparatus 100 may acquire a movingimage from a source other than the broadcast wave. For example, themoving image supply apparatus 100 may acquire the moving image byreading out a moving image recorded on a recording medium such as a DVD(Digital Versatile Disk) or recorded in a storage device such as an HDD(Hard Disk Drive).

The image processing apparatus 200 converts the frame rate of the movingimage in the input signal and outputs an output signal, which containsthe moving image whose frame rate is converted, to the display apparatus400 via a signal line 209. The frame rate of the moving image in theoutput signal is 120 hertz, for example.

In general, when a ratio of the frame rate of the input signal to theframe rate of the output signal is 1:m (m is a real number largerthan 1) in a frame rate conversion, such a conversion is referred to asan mX speed frame rate conversion. For example, the conversion from 60hertz to 120 hertz is a double-speed frame rate conversion. Note thatthe image processing apparatus 200 may perform a frame rate conversionsuch as a quadruple-speed frame rate conversion from 30 hertz to 120hertz, in addition to the double-speed frame rate conversion.

The display apparatus 400 reproduces and displays the moving imagecontained in the output signal.

(Configuration Example of Image Processing Apparatus)

FIG. 2 is a block diagram showing a configuration example of the imageprocessing apparatus 200 in the first embodiment. The image processingapparatus 200 includes image memories 210, 220, and 230, a selector 240,and an interpolated image generation unit 300.

The image memories 210 and 230 each store an original image F_(n)contained in the input signal. Herein, n is an integer indicating theorder in which original images are reproduced. The interpolated imagegeneration unit 300 and the selector 240 read out the original imagesstored in the image memories 210 and 230, respectively, when the nextoriginal image is input, so that an original image delayed by one imagecan be acquired.

The interpolated image generation unit 300 generates, based on theoriginal images F_(n) and F_(n-1), an image to be interpolated betweenthose original images F_(n) and F_(n-1). The generated image is acombined interpolated image MC_mix. The interpolated image generationunit 300 acquires the original image F_(n) from the input signal andacquires the delayed original image F_(n-1) from the image memory 210,to generate a combined interpolated image MC_mix based on those originalimages and cause the image memory 220 to store the resultant imageMC_mix. The image memory 220 is for storing the combined interpolatedimage MC_mix.

The selector 240 selects and outputs the combined interpolated imageMC_mix stored in the image memory 220 and the original image stored inthe image memory 230 in a time-series order.

(Configuration Example of Interpolated Image Generation Unit)

FIG. 3 is a block diagram showing a configuration example of theinterpolated image generation unit 300 in the first embodiment. Theinterpolated image generation unit 300 includes a local motion vectordetection unit 310, a normal interpolated image generation unit 320, ahigh-frequency area extraction unit 330, a global motion vectordetection unit 340, a high-frequency area interpolated image generationunit 350, a combination ratio determination unit 360, and a combinationunit 370.

The local motion vector detection unit 310 divides the original imageinto a plurality of blocks each having a predetermined shape and detectsa vector, i.e., a local motion vector, which indicates a direction and adistance in which each of those blocks moves within a certain period oftime. Specifically, the local motion vector detection unit 310 handlesthe original image as an image formed of square blocks each having 8×8pixels, for example. The local motion vector detection unit 310 selectsany one of the blocks of an interpolated image to use the block as atarget block. The local motion vector detection unit 310 detects a blockof the original image, which corresponds to the target block of theinterpolated image, and then sets a search range W for an area in theoriginal image, which falls within a certain range from the block of theoriginal image. The search range W is a range used for searching for alocal motion vector by block matching. For example, in the originalimage, the search range W is set for the target block and apredetermined number of blocks around the target block.

Here, a vector that is a candidate when a local motion vector issearched for is referred to as a candidate motion vector mv. Thecandidate motion vector my can be divided into v_(n) and v_(n-1) byExpressions 1 and 2 below.

$\begin{matrix}\begin{matrix}{V_{n} = ( {v_{x\_ n},v_{y\_ n}} )} \\{= ( {{{- {mvX}} \cdot ( {1 - {Relpos}} )},{{- {mvY}} \cdot ( {1 - {Relpos}} )}} )}\end{matrix} & {{Expression}\mspace{14mu} 1} \\\begin{matrix}{V_{n - 1} = ( {V_{{x\_ n} - 1},v_{{y\_ n} - 1}} )} \\{= ( {{{mvX} \cdot {Relpos}},{{mvY} \cdot {RelPos}}} )}\end{matrix} & {{Expression}\mspace{14mu} 2}\end{matrix}$

In Expressions 1 and 2, mvX represents an x component of the candidatemotion vector mv, and mvY represents a y component of the candidatemotion vector mv. Further, in Expressions 1 and 2, Relpos represents aninterpolation position parameter that indicates an interpolationposition. Here, the interpolation position is a reproduction position ofeach combined interpolated image MC_mix in the output signal on a timeaxis on which the moving image is reproduced. The interpolation positionparameter or Relpos is a value obtained by dividing a time between theoriginal image and the interpolated image by a time between two adjacentoriginal images, for example. In the double-speed frame rate conversion,the time between the original image and the interpolated image is a halfof the time between the two adjacent original images, and theinterpolation position parameter of Relpos is “0.50”. Additionally, inthe quadruple-speed frame rate conversion, three interpolated images areinserted between two adjacent original images, and the time between oneof the original images and each interpolated image is ¼, ½, or ¾ of thetime between the two original images. Consequently, the interpolationposition parameter of Relpos of each of the interpolated images is“0.25”, “0.50”, or “0.75”.

When the candidate motion vector my passes through a pixel atcoordinates (x, y) in the target block, a pixel value “succ” of a pixelcorresponding to an end point of the candidate motion vector my in theoriginal image F_(n) is expressed by Expression 3 below. Further, apixel value “prev” of a pixel corresponding to a start point of thecandidate motion vector my in the original image F_(n-1) is expressed byExpression 4 below.

succ=F _(n)(x+v _(x) _(—) _(n) ,y+v _(y) _(—) _(n))  Expression 3

prev=F _(n-1)(x+v _(x) _(—) _(n-1) ,y+v _(y) _(—) _(n-1))  Expression 4

In Expression 3, F_(n)(x, y) represents a pixel value of the pixel atcoordinates (x, y) in the original image F_(n). In Expression 4, F_(n-1)(x, y) represents a pixel value of the pixel at coordinates (x, y) inthe original image F_(n-1).

Within the search range W, statistics of differences between the pixelvalues “prev” and the pixel values “succ” are obtained by Expressions 3and 4, to obtain two blocks having the highest correlation within thesearch range W. The local motion vector detection unit 310 detects acandidate motion vector extending from one of the two blocks having thehighest correlation to the other block and sets the detected vector as alocal motion vector. Specifically, the local motion vector detectionunit 310 detects a candidate motion vector my that satisfies Expressions5 and 6 and sets the detected candidate motion vector my as a localmotion vector mv_(L).

$\begin{matrix}{{E_{SAD}({mv})} = {\sum\limits_{x \in B}^{\;}\; {\sum\limits_{y \in B}^{\;}\; {{{prev} - {succ}}}}}} & {{Expression}\mspace{14mu} 5} \\{{mv} = {\arg \; {\min\limits_{{mv} \in W}\mspace{14mu} {E_{SAD}({mv})}}}} & {{Expression}\mspace{14mu} 6}\end{matrix}$

In Expression 5, x and y represent coordinates of a pixel in the targetblock. E_(SAD) represents the sum of absolute difference values ofvalues of corresponding pixels of the two blocks and is used as anevaluation value indicating the level of the correlation between thoseblocks. Further, B represents the target block. In Expression 6,minE_(SAD) represents a minimum value of E_(SAD). argminE_(SAD)(mv)represents a function that returns my obtained when E_(SAD) takes aminimum value, that is, when the correlation is highest.

As exemplified in Expressions 5 and 6, the processing of evaluating thelevel of the correlation based on the sum of absolute difference valuesof pixel values is referred to as SAD (Sum of Absolute Differenceestimation). Note that the local motion vector detection unit 310 canalso obtain the level of the correlation by processing other than theSAD. For example, the local motion vector detection unit 310 may use MPC(Maximum matching Pixel Count). In the MPC, the number of pixels atwhich an absolute difference value of pixel values is equal to orsmaller than a predetermined matching threshold is counted to evaluatethe correlation level based on the count value. In the MPC, a candidatemotion vector obtained when the count value is maximum is detected asthe local motion vector.

The local motion vector detection unit 310 obtains the local motionvector mv_(L) for each of all the blocks of the interpolated image andobtains a forward motion vector v_(n) _(—) _(L) and a backward motionvector v_(n-1) _(—) _(L) by using Expressions 7 and 8.

$\begin{matrix}\begin{matrix}{V_{n\_ L} = ( {v_{{x\_ n}{\_ L}},v_{{y\_ n}{\_ L}}} )} \\{= ( {{{- {mvX}_{L}} \cdot ( {1 - {Relpos}} )},{{- {mvY}_{L}} \cdot ( {1 - {Relpos}} )}} )}\end{matrix} & {{Expression}\mspace{14mu} 7} \\\begin{matrix}{V_{n - {1{\_ L}}} = ( {V_{{x\_ n} - {1{\_ L}}},v_{{y\_ n} - {1{\_ L}}}} )} \\{= ( {{{mvX}_{L} \cdot {Relpos}},{{mvY}_{L} \cdot {RelPos}}} )}\end{matrix} & {{Expression}\mspace{14mu} 8}\end{matrix}$

In Expressions 7 and 8, mvX_(L) is an x component of the local motionvector mv_(L) and mvY_(L) is a y component of the local motion vectormv_(L).

The local motion vector detection unit 310 supplies the forward motionvector v_(n) _(—) _(L) and the backward motion vector v_(n-1) _(—) _(L),of each block to the normal interpolated image generation unit 320.

The normal interpolated image generation unit 320 generates an image tobe interpolated between the original images F_(n) and F_(n-1), which isa normal interpolated image, based on the local motion vector mv_(L) andthe original images F_(n) and F_(n-1). The normal interpolated imagegeneration unit 320 receives the forward motion vector v_(n) _(—) _(L)and the backward motion vector v_(n-1) _(—) _(L) from the local motionvector detection unit 310. The normal interpolated image generation unit320 substitutes the received vectors into Expressions 9 and 10 below toobtain the pixel values “succ” and “prev” in the original images F_(n)and F_(n-1).

succ=F _(n)(x+v _(x) _(—) _(n) _(—) _(L) ,y+v _(y) _(—) _(n) _(—)_(L))  Expression 9

prev=F _(n-1)(x+v _(x) _(—) _(n-1) _(—) _(L) ,y+v _(y) _(—) _(n-1) _(—)_(L))  Expression 10

The normal interpolated image generation unit 320 substitutes the pixelvalues acquired by Expressions 9 and 10 into Expression 11 below tocalculate a pixel value “out”.

out=prev·(1−Relpos)+succ·Relpos  Expression 11

The normal interpolated image generation unit 320 supplies an imageformed of the pixel values “out”, which is a normal interpolated imageMC_N, to the combination unit 370.

The high-frequency area extraction unit 330 extracts an area having aspatial frequency higher than a predetermined value from each of theoriginal images F_(n) and F_(n-1), and sets the extracted areas ashigh-frequency areas HP_(n) and HP_(n-1). Further, the high-frequencyarea extraction unit 330 detects a difference value k in pixel valuebetween the original images F_(n) and F_(n-1) for each pixel.Subsequently, the high-frequency area extraction unit 330 amplifies eachpixel value of the extracted high-frequency areas HP_(n) and HP_(n-1) inaccordance with the difference value k, to generate high-frequency areasHPK_(n) and HPK_(n-1). The high-frequency area extraction unit 330supplies those high-frequency areas HPK_(n) and HPK_(n-1) to the globalmotion vector detection unit 340 and the combination ratio determinationunit 360.

The global motion vector detection unit 340 detects a vector, i.e., aglobal motion vector, which indicates a direction and a distance inwhich the high-frequency area moves within a certain period of time. Theglobal motion vector detection unit 340 receives the high-frequencyareas HPK_(n) and HPK_(n-1) from the high-frequency area extraction unit330 and detects the vector indicating the moving direction of the wholeof those areas, to set the detected vector as a global motion vectormv_(G). Specifically, the global motion vector detection unit 340 setsthe search range W for the whole of the high-frequency areas HPK_(n) andHPK_(n-1) and detects a candidate motion vector that satisfiesExpressions 6 and 12 below as the global motion vector mv_(G).

$\begin{matrix}{{E_{SAD}({mv})} = {\sum\limits_{x \in {MC}}^{\;}\; {\sum\limits_{y \in {MC}}^{\;}\; {{{prev} - {succ}}}}}} & {{Expression}\mspace{14mu} 12}\end{matrix}$

In Expression 12, MC represents the entire interpolated image.

The global motion vector detection unit 340 substitutes the obtainedglobal motion vector mv_(G) into Expressions 13 and 14 below and obtainsa forward motion vector V_(n) _(—) _(G) and a backward motion vectorV_(n-1) _(—) _(G).

$\begin{matrix}\begin{matrix}{V_{n\_ G} = ( {v_{{x\_ n}{\_ G}},v_{{y\_ n}{\_ G}}} )} \\{= ( {{{- {mvX}_{G}} \cdot ( {1 - {Relpos}} )},{{- {mvY}_{G}} \cdot ( {1 - {Relpos}} )}} }\end{matrix} & {{Expression}\mspace{14mu} 13} \\\begin{matrix}{V_{n - {1{\_ G}}} = ( {V_{{x\_ n} - {1{\_ G}}},v_{{y\_ n} - {1{\_ G}}}} )} \\{= ( {{{mvX}_{G} \cdot {Relpos}},{{mvY}_{G} \cdot {RelPos}}} )}\end{matrix} & {{Expression}\mspace{14mu} 14}\end{matrix}$

In Expressions 13 and 14, mvX_(G) represents an x component of theglobal motion vector mv_(G) and mvY_(G) represents a y component of theglobal motion vector mv_(G).

The global motion vector detection unit 340 supplies the forward motionvector V_(n) _(—) _(c) and the backward motion vector V_(n-1) _(—) _(G)to the high-frequency area interpolated image generation unit 350 andthe combination ratio determination unit 360. Note that the globalmotion vector detection unit 340 is an example of a vector detectionunit in the section “What is claimed is”.

The high-frequency area interpolated image generation unit 350 generatesan image to be interpolated between the original images F_(n) andF_(n-1), which is a high-frequency area interpolated image, based on theglobal motion vector mv_(G) and the original images F_(n) and F_(n-1).The high-frequency area interpolated image generation unit 350 receivesthe forward motion vector v_(n) _(—) _(G) and the backward motion vectorV_(n-1) _(—) _(G) from the global motion vector detection unit 340. Thehigh-frequency area interpolated image generation unit 350 substitutesthe received vectors into Expressions 15 and 16 below to obtain thepixel values “succ” and “prev” in the original images F_(n) and F_(n-1).

succ=F _(n)(x+v _(x) _(—) _(n) _(—) _(G) ,y+v _(y) _(—) _(n) _(—)_(G))  Expression 15

prev=F _(n-1)(x+v _(x) _(—) _(n-1) _(—) _(G) ,y+v _(y) _(—) _(n-1) _(—)_(G))  Expression 16

The high-frequency area interpolated image generation unit 350substitutes the pixel values acquired by Expressions 15 and 16 intoExpression 11 to calculate a pixel value “out”. The high-frequency areainterpolated image generation unit 350 supplies an image formed of thepixel values “out”, which is a high-frequency area interpolated imageMC_HP_(n) to the combination unit 370.

The combination ratio determination unit 360 determines a combinationratio of the normal interpolated image MC_N and the high-frequency areainterpolated image MC_HP. The combination ratio determination unit 360receives the high-frequency areas HPK_(n) and HPK_(n-1) from thehigh-frequency area extraction unit 330 and combines the high-frequencyareas HPK_(n) and HPK_(n-1) to generate combination ratio data MIX. Thecombination ratio data MIX is image data that indicates the combinationratio of each pixel by pixel values. The combination ratio determinationunit 360 supplies the combination ratio data MIX to the combination unit370.

The combination unit 370 combines the normal interpolated image MC_N andthe high-frequency area interpolated image MC_HP based on thecombination ratio data MIX. The combination unit 370 causes the imagememory 220 to store the combined image as the combined interpolatedimage MC_mix.

FIGS. 4A, 4B, and 4C are each a diagram for describing a method ofdetecting a local motion vector in the first embodiment. FIG. 4A is anexample of an original image 510. FIG. 4B is an example of an originalimage 520 subsequent to the original image 510.

In the normal interpolated image, it is assumed that a block located atthe same position as a block 511 in the original image 510 is selectedas a target block. In this case, in the original image 510, the block511 and a predetermined number of blocks around the block 511 (forexample, 8 blocks) are set as a search range 512. Further, also in theoriginal image 520, a block 521 corresponding to the block 511 and apredetermined number of blocks around the block 521 (for example, 8blocks) are set as a search range 522. Subsequently, block matchingbetween the blocks in the search range 512 and the blocks in the searchrange 522 is performed by using Expressions 5 and 6 and the like.

It is assumed that the following result is obtained in the blockmatching: a block 513 on the upper right of the search range 512 and ablock 523 on the lower left of the search range 522 have the highestcorrelation. From this result, the vector indicating a direction and adistance from the block 513 to the block 523 is detected as a localmotion vector 524.

FIG. 4C is a diagram showing an example of the forward motion vector andthe backward motion vector. The local motion vector 524 is divided intothe forward motion vector V_(n) _(—) _(L) and the backward motion vectorV_(n-1) _(—) _(L) by Expressions 7 and 8.

FIG. 5 is a diagram showing an example of the local motion vector in thefirst embodiment. The local motion vector is obtained for each block,and each local motion vector is divided into the forward motion vectorV_(n) _(—) _(L) and the backward motion vector V_(n-1) _(—) _(L). As aresult, as exemplified in FIG. 5, the forward motion vector V_(n) _(—)_(L) and the backward motion vector V_(n-1) _(—) _(L) are obtained foreach block. For example, when a local motion vector (8, 0) is detectedin a block B10 in the normal interpolated image, this vector is dividedinto a forward motion vector (—4, 0) and a backward motion vector (4, 0)by Expressions 1 and 2.

(Configuration Example of Normal Interpolated Image Generation Unit)

FIG. 6 is a block diagram showing a configuration example of the normalinterpolated image generation unit 320 in the first embodiment. Thenormal interpolated image generation unit 320 includes a forward motioncompensation unit 321, a backward motion compensation unit 322, an adder323, and a multiplier 324.

The forward motion compensation unit 321 executes forward motioncompensation processing to acquire each pixel value of coordinates (x,y) in the normal interpolated image based on the forward motion vectorv_(n) _(—) _(L). Specifically, the pixel value “succ” in the originalimage F_(n), which is obtained by Expression 9, is acquired as a pixelvalue at the coordinates (x, y) in the normal interpolated image. Theforward motion compensation unit 321 supplies the acquired pixel valueto the adder 323.

The backward motion compensation unit 322 executes backward motioncompensation processing to acquire each pixel value of coordinates (x,y) in the normal interpolated image based on the backward motion vectorV_(n-1) _(—) _(L). Specifically, the pixel value “prev” in the originalimage F_(n-1), which is obtained by Expression 10, is acquired as apixel value at the coordinates (x, y) in the normal interpolated image.The backward motion compensation unit 322 supplies the acquired pixelvalue to the adder 323.

The adder 323 adds the input values. The adder 323 receives an input ofthe pixel value “succ” acquired by the forward motion compensation unit321 and an input of the pixel value “prev” acquired by the backwardmotion compensation unit 322. The adder 323 adds those input values andsupplies the resultant value to the multiplier 324.

The multiplier 324 multiplies the input value by a predetermined value.The multiplier 324 receives an input of the additional value suppliedfrom the adder 323. The multiplier 324 multiplies the input value by 0.5and supplies the resultant value, which is the pixel value “out” in thenormal interpolated image MC_N, to the combination unit 370.

The adder 323 and the multiplier 324 implement an arithmetic operationof Expression 11 when the interpolation position parameter of Relpos is0.5.

Note that the normal interpolated image generation unit 320 executesboth of the forward motion compensation processing and the backwardmotion compensation processing, but it may execute any one of theforward motion compensation processing and the backward motioncompensation processing. In this case, the normal interpolated imagegeneration unit 320 includes one of the forward motion compensation unit321 and the backward motion compensation unit 322 and does not includethe other motion compensation unit, the adder 323, and the multiplier324. Subsequently, a pixel value generated by any one of the forwardmotion compensation unit 321 and the backward motion compensation unit322 is output without change to the combination unit 370, to serve asthe pixel value of the normal interpolated image.

FIGS. 7A, 7B, and 7C are each a diagram for describing a method ofgenerating a normal interpolated image in the first embodiment. FIG. 7Ais an example of a forward motion compensation image 530 generated bythe forward motion compensation unit 321. The forward motioncompensation unit 321 uses Expression 9 to acquire, from the originalimage F_(n), a pixel value “succ” of a pixel that is located at aposition to which a pixel at coordinates (x, y) in a target block 541 ofthe forward motion compensation image 530 is moved along the forwardmotion vector V_(n) _(—) _(L). This pixel value “succ” is used as apixel value of the pixel at coordinates (x, y) in the target block 541.In other words, the forward motion compensation image 530 is generatedusing a pixel obtained by moving the pixel of the original image F_(n)according to the forward motion vector V_(n) _(—) _(L).

FIG. 7B is an example of a backward motion compensation image 540generated by the backward motion compensation unit 322. The backwardmotion compensation unit 322 uses Expression 10 to acquire, from theoriginal image F_(n-1), a pixel value “prev” of a pixel that is locatedat a position to which a pixel at coordinates (x, y) in a target block531 of the backward motion compensation image 540 is moved along thebackward motion vector V_(n-1) _(—) _(L). This pixel value “prev” isused as a pixel value of the pixel at coordinates (x, y) in the targetblock 541. In other words, the backward motion compensation image 540 isgenerated using a pixel obtained by moving the pixel of the originalimage F_(n-1) according to the backward motion vector V_(n-1) _(—) _(L).

FIG. 7C is a diagram showing an example of a normal interpolated image550 generated by combining the forward motion compensation image 530shown in FIG. 7A and the backward motion compensation image 540 shown inFIG. 7B based on Expression 11. The forward motion vector V_(n) _(—)_(L) and the backward motion vector V_(n-1) _(—) _(L) are vectorsobtained when the local motion vector mv_(L) is divided. Consequently,images in which pixels are moved according to those forward motionvector V_(n) and backward motion vector V_(n-1) are combined to eachother, and a block located between the blocks moved according to thelocal motion vector mv_(L) is interpolated.

(Configuration Example of High-Frequency Area Extraction Unit)

FIG. 8 is a block diagram showing a configuration example of thehigh-frequency area extraction unit 330 in the first embodiment. Thehigh-frequency area extraction unit 330 includes a low-pass filter 331,a subtracter 332, a multiplier 333, an image memory 334, and adifference detection unit 335.

The low-pass filter 331 extracts a low-frequency area LP_(n) from theoriginal image F_(n), the low-frequency area LP_(n) having a spatialfrequency lower than a predetermined value. The low-pass filter 331 isachieved by a FIR (Finite Impulse Response) filter, for example. Thelow-pass filter 331 supplies a pixel value of each pixel in theextracted low-frequency area LP_(n) to the subtracter 332.

The subtracter 332 subtracts one of two input values from the othervalue. The subtracter 332 sequentially receives an input of the pixelvalue in the low-frequency area LP_(n) extracted by the low-pass filter331 and an input of the pixel value in the original image F_(n). Thesubtracter 332 subtracts the pixel value in the low-frequency areaLP_(n) from the pixel value in the original image F_(n) and supplies theresultant value, which is a pixel value of the high-frequency areaHP_(n), to the multiplier 333.

Note that the high-frequency area extraction unit 330 extracts thehigh-frequency area HP_(n) by the low-pass filter 331 and the subtracter332, but the high-frequency area extraction unit 330 may include ahigh-pass filter instead of the low-pass filter 331 and the subtracter332 and extract the high-frequency area HP_(n) by performing high-passfiltering on the original image F_(n). Further, the low-pass filter 331and the subtracter 332 are examples of a filter unit in the section“What is claimed is”.

The difference detection unit 335 detects a difference in pixel valuebetween the original images F_(n) and F_(n-1) for each pixel. Thedifference detection unit 335 detects a difference value k for eachpixel. The difference detection unit 335 generates difference data Kformed of those difference values k. The difference detection unit 335sequentially supplies each of the difference values k in the differencedata K to the multiplier 333.

The multiplier 333 multiplies the input values. The multiplier 333receives an input of the difference value k in the difference data K_(n)and an input of the pixel value in the high-frequency area HP_(n). Themultiplier 333 multiplies those input values and supplies the resultantvalue, which is a pixel value in the high-frequency area HPK_(n), to theglobal motion vector detection unit 340 and the combination ratiodetermination unit 360. Further, the multiplier 333 stores themultiplied value in the image memory 334. Note that the multiplier 333is an example of an increase unit in the section “What is claimed is”.

The multiplier 333 multiplies the pixel value of the high-frequency areaHP_(n) by the difference value k, so that the pixel value of thehigh-frequency area that moves between the original images F_(n) andF_(n-1) is amplified. Consequently, the global motion vector detectionunit 340 can easily detect a vector, which indicates a moving directionof the high-frequency area with motion, as a global motion vector.Further, since the pixel value of the high-frequency area with motion isamplified, the combination ratio of the high-frequency area with motioncan be preferentially increased.

The image memory 334 stores the high-frequency area. The global motionvector detection unit 340 and the combination ratio determination unit360 read out the high-frequency area stored in the image memory 334 whenthe next high-frequency area is generated, so that a delayedhigh-frequency area HPK_(n-1) can be acquired.

(Configuration Example of Difference Detection Unit)

FIG. 9 is a block diagram showing a configuration example of thedifference detection unit 335 in the first embodiment. The differencedetection unit 335 includes a subtracter 336, an absolute valueacquisition unit 337, and a difference adjustment filter 338.

The subtracter 336 subtracts one of the two input values from the othervalue. The subtracter 336 sequentially receives an input of a pixelvalue of a pixel in the original image F_(n) and an input of a pixelvalue of a pixel in the original image F_(n-1). The subtracter 336subtracts one of those input values from the other value and suppliesthe resultant value, i.e., a difference of the pixel values, to theabsolute value acquisition unit 337.

The absolute value acquisition unit 337 acquires an absolute value ofthe input value. The absolute value acquisition unit 337 sequentiallyreceives an input of the difference supplied from the subtracter 336.The absolute value acquisition unit 337 supplies data formed of absolutevalues diff of the differences, which serves as difference absolutevalue data DIF_(n) to the difference adjustment filter 338.

The difference adjustment filter 338 adjusts each absolute value diff inthe difference absolute value data DIF. For example, each time anabsolute value is input, the difference adjustment filter 338 uses thefollowing expression to calculate a value obtained by amplifying theabsolute value, and sets the resultant value as a difference value k.The difference adjustment filter 338 outputs data formed of thosedifference values k, as difference data K, to the multiplier 333.

$\quad\begin{matrix}\{ \begin{matrix}{k = 0} & ( {{diff} \leqq {Th}_{d\; 1}}  \\{k = {{a \times {diff}} + b}} & ( {{Th}_{d\; 1} < {diff} < {Th}_{d\; 2}} ) \\{k = 255} & ( {{diff} \geqq {Th}_{d\; 2}} )\end{matrix}  & {{Expression}\mspace{14mu} 17}\end{matrix}$

In Expression 17, “a” and “b” are factors, each of which is a realnumber. Further, Th_(d1) is a threshold larger than 0 and smaller thanTh_(d2), and Th_(d2) is a threshold larger than Th_(d1) and smaller than255.

Note that Expression 17 assumes that the pixel value has a valueexpressed in 8 bits, specifically, a value ranging from 0 to 255, butthe number of bits assigned to the pixel value is not limited to 8 bits.For example, the pixel value may be expressed in 10 bits. In such acase, in Expression 17, “255” is replaced with “1023”.

The difference adjustment filter 338 amplifies the absolute value, sothat the pixel value of the high-frequency area with motion isincreased. Consequently, the global motion vector can be easily detectedand the combination ratio of the high-frequency area with motion ispreferentially set to be high.

Note that the difference adjustment filter 338 amplifies the absolutevalue of the difference, but the difference adjustment filter 338 mayoutput the absolute value as the difference value k without performingamplification. Further, the difference adjustment filter 338 mayattenuate the absolute value. When the absolute value of the differenceis extremely large, there is a possibility that motion of a noisecomponent is erroneously detected as a global motion vector. In thiscase, the attenuation of the absolute value suppresses an occurrence ofa false detection. Additionally, when the absolute value of thedifference is extremely large, there is a possibility that aninappropriate combination ratio is determined based on the noisecomponent. In this case, the attenuation of the absolute value allows anappropriate combination ratio to be determined.

(Configuration Example of High-Frequency Area Interpolated ImageGeneration Unit)

FIG. 10 is a block diagram showing a configuration example of thehigh-frequency area interpolated image generation unit 350 in the firstembodiment. The high-frequency area interpolated image generation unit350 includes a forward motion compensation unit 351, a backward motioncompensation unit 352, an adder 353, and a multiplier 354.

The configuration of the forward motion compensation unit 351 is thesame as that of the forward motion compensation unit 321 in the normalinterpolated image generation unit 320 except that the forward motioncompensation processing is executed based on the forward motion vectorv_(n) _(—) _(G) instead of the forward motion vector V_(n) _(—) _(L).The forward motion compensation unit 351 supplies each pixel value“succ” acquired by the forward motion compensation processing to theadder 353.

The configuration of the backward motion compensation unit 352 is thesame as that of the backward motion compensation unit 322 in the normalinterpolated image generation unit 320 except that the backward motioncompensation processing is executed based on the backward motion vectorv_(n-1) _(—) _(G) instead of the backward motion vector V_(n-1) _(—)_(L). The backward motion compensation unit 352 supplies each pixelvalue “prev” acquired by the backward motion compensation processing tothe adder 353.

The adder 353 adds the input values. The adder 353 receives an input ofthe pixel value “succ” acquired by the forward motion compensation unit351 and an input of the pixel value “prev” acquired by the backwardmotion compensation unit 352. The adder 353 adds those input values andsupplies the resultant value to the multiplier 354.

The multiplier 354 multiplies the input value by a predetermined value.The multiplier 354 receives an input of the additional value suppliedfrom the adder 353. The multiplier 354 multiplies the input value by 0.5and supplies the resultant value, which is the pixel value “out” in thehigh-frequency area interpolated image MC_HP, to the combination unit370.

Note that the high-frequency area interpolated image generation unit 350executes both of the forward motion compensation processing and thebackward motion compensation processing, but it may execute any one ofthe forward motion compensation processing and the backward motioncompensation processing. In this case, the high-frequency areainterpolated image generation unit 350 includes one of the forwardmotion compensation unit 351 and the backward motion compensation unit352 and does not include the other motion compensation unit, the adder353, and the multiplier 354. Subsequently, a pixel value generated byany one of the forward motion compensation unit 351 and the backwardmotion compensation unit 352 is output without change to the combinationunit 370, to serve as the pixel value of the normal interpolated image.

(Configuration Example of Combination Ratio Determination Unit)

FIG. 11 is a block diagram showing a configuration example of thecombination ratio determination unit 360 in the first embodiment. Thecombination ratio determination unit 360 includes a forward motioncompensation unit 361, a backward motion compensation unit 362, amultiplier 363, a gain adjustment unit 364, and an interpolation filter365.

The configuration of the forward motion compensation unit 361 is thesame as that of the forward motion compensation unit 351 in thehigh-frequency area interpolated image generation unit 350 except thatthe forward motion compensation processing is executed based on thehigh-frequency area HPK_(n) instead of the original image F_(n). Theforward motion compensation unit 361 supplies each pixel value “succ”acquired by the forward motion compensation processing to the multiplier363.

The configuration of the backward motion compensation unit 362 is thesame as that of the backward motion compensation unit 352 in thehigh-frequency area interpolated image generation unit 350 except thatthe backward motion compensation processing is executed based on thehigh-frequency area HPK_(n-1) instead of the original image F_(n-1). Thebackward motion compensation unit 362 supplies each pixel value “prev”acquired by the backward motion compensation processing to themultiplier 363.

The multiplier 363 multiplies the input values. The multiplier 363receives an input of the pixel value “succ” acquired by the forwardmotion compensation unit 361 and an input of the pixel value “prev”acquired by the backward motion compensation unit 362. The multiplier363 multiplies those input values and supplies the resultant value tothe gain adjustment unit 364.

The multiplier 363 multiplies the pixel value “succ” and the pixel value“prev”, so that the pixel value of the high-frequency area is amplifiedmore than when the input values are added. Consequently, the combinationratio of the high-frequency area is preferentially set to be high.

The gain adjustment unit 364 adjusts a gain and attenuates the pixelvalue supplied from the multiplier 363. The gain adjustment unit 364attenuates the pixel value from the multiplier 363 such that the valuefalls within a certain range through the gain adjustment. For example,when a pixel value has the data size of 8 bits and such pixel values aremultiplied, the data size of the resultant value after themultiplication is 16 bits. The multiplication result is attenuated suchthat its data size is changed to 8 bits. The gain adjustment unit 364supplies the adjusted pixel value to the interpolation filter 365.

The interpolation filter 365 interpolates, in an image MC_M formed ofthe pixel values generated by the gain adjustment unit 364, a portionwhere the edge interrupts, to generate the combination ratio data MIX.The interpolation filter 365 uses the following expression to performinterpolation, for example. The interpolation filter 365 supplies thegenerated combination ratio data MIX to the combination unit 370.

MIX(x,y)=max{MC _(—) M(x−1,y)MC _(—) M(x,y)MC _(—) M(x+1,y)}  Expression18

In Expression 18, MC_M(x, y) represents a pixel value of a pixel atcoordinates (x, y) in the image MC_M. MIX(x, y) represents a pixel valueof a pixel at coordinates (x, y) in the combination ratio data MIX. max() represents a maximum value of the pixel value in parentheses.

(Configuration Example of Combination Unit)

FIG. 12 is a block diagram showing a configuration example of thecombination unit 370 in the first embodiment. The combination unit 370includes a level adjustment unit 371, a multiplier 372, a subtracter373, a multiplier 374, and an adder 375.

The level adjustment unit 371 adjusts the pixel value in the combinationratio data MIX so as to fall within the range from 0 to 1. For example,when the pixel value takes a value within the range from 0 to 255, thelevel adjustment unit 371 divides the pixel value by 255 to adjust theresultant value to a value falling within the range from 0 to 1. Thelevel adjustment unit 371 sequentially supplies each of the adjustedvalues, which is a combination ratio mix, to the subtracter 373 and themultiplier 372.

The multiplier 372 multiplies the input values. The multiplier 372receives an input of the combination ratio mix supplied from the leveladjustment unit 371 and an input of the pixel value in thehigh-frequency area interpolated image MC_HP. The multiplier 372multiplies those input values and supplies the resultant value to theadder 375.

The subtracter 373 subtracts the input value from a predetermined value.The subtracter 373 receives an input of the combination ratio mixsupplied from the level adjustment unit 371. The subtracter 373subtracts the combination ratio mix from 1.0 and supplies the resultantvalue (1.0-mix) to the multiplier 374.

The multiplier 374 multiplies the input values. The multiplier 374receives an input of the value (1.0-mix) supplied from the subtracter373 and an input of the pixel value in the normal interpolated imageMC_N. The multiplier 374 multiplies those input values and supplies theresultant value to the adder 375.

The adder 375 adds the input values. The adder 375 receives inputs ofthe multiplication results of the respective multipliers 372 and 374.The adder 375 adds those input values and stores, in the image memory220, the resultant value serving as a pixel value in the combinedinterpolated image MC_mix.

To sum it up, the multiplier 372, the subtracter 373, the multiplier374, and the adder 375 execute the following arithmetic operation ofExpression 19 below.

MC_mix(x,y)=MC _(—) N(x,y)×(1.0−mix)+MC _(—) HP(x,y)×mix  Expression 19

In Expression 19, MC_mix(x, y) represents a pixel value of a pixel atcoordinates (x, y) in the combined interpolated image MC_mix. MC_N(x, y)represents a pixel value of a pixel at coordinates (x, y) in the normalinterpolated image MC_N. MC_HP(x, y) represents a pixel value of a pixelat coordinates (x, y) in the high-frequency area interpolated imageMC_HP. mix represents a combination ratio generated by the leveladjustment unit 371 based on the pixel value at coordinates (x, y) inthe combination ratio data MIX.

FIG. 13 is a diagram showing an example of a normal interpolated image610 in the first embodiment. Here, it is assumed that the imageprocessing apparatus 200 receives an input of a moving image in which acharacter telop, which moves in a horizontal direction with the lapse oftime, is displayed on a lower portion of a screen. The image processingapparatus 200 generates the normal interpolated image 610 from theoriginal images F_(n) and F_(n-1) in the moving image. The normalinterpolated image 610 includes a character telop 611 in its lowerportion. Note that characters are also displayed above the charactertelop 611, but those character portions do not move with the lapse oftime.

As described above, the image processing apparatus 200 detects a localmotion vector for each block and moves the block in the original imagebased on the local motion vector, to generate the normal interpolatedimage 610. As to the area of the character telop, the entire area movesin the horizontal direction, and thus all the values of the local motionvectors in the character telop are thought to be the same.

When a lot of blocks exist in the character telop, however, there is acase where a vector with a value, which is different from values oflocal motion vectors of adjacent blocks, may be detected from someblocks in the character telop. In this case, the local motion vector ofsuch a block does not have a correct value, and it is evaluated that thelocal motion vector detection has failed.

A movement amount of the block for which the local motion vectordetection has failed is different from an actual movement amount of thecharacter telop, and thus a blur occurs in a boundary between the blockfor which the detection has failed and a block for which the detectionhas not failed. For example, in the character telop 611 in the normalinterpolated image 610, blurs occur at a plurality of portions such asblurs 612 and 613.

FIG. 14 is a diagram showing an example of a high-frequency areainterpolated image 620 in the first embodiment. The image processingapparatus 200 moves the entire original image according to the globalmotion vectors each indicating the movement of a character telop 621 andgenerates the high-frequency area interpolated image 620, so that noblurs occur in the character telop 621. On the other hand, since theentire original image is moved according to the global motion vectorsirrespective of no movements at portions other than the character telop621, blurs occur at the portions other than the character telop 621.

FIG. 15 is a diagram showing an example of combination ratio data 630 inthe first embodiment. The combination ratio data 630 is image dataincluding, in the high-frequency area with motion, a pixel whose pixelvalue is amplified. For that reason, a pixel value of the charactertelop portion extracted as a high-frequency area with motion is higherthan those of the other portions. Consequently, the value of thecombination ratio mix in the character telop portion is set to be higherthan those of the other portions.

FIG. 16 is a diagram showing an example of a combined interpolated image640 in the first embodiment. The image processing apparatus 200 combinesthe normal interpolated image 610 and the high-frequency areainterpolated image 620 based on the combination ratio data 630. In thecombination ratio data 630, the combination ratio of the character telopportion is set to be higher. Consequently, a combined interpolated image640 in which the ratio of the high-frequency area interpolated image 620is higher in a character telop 631 and the ratio of the normalinterpolated image 610 is higher in the other portions is generated. Noblurs occur in the character telop in the high-frequency areainterpolated image 620, and no blurs occur in portions other than thecharacter telop in the normal interpolated image 610. Consequently, noblurs occur as a whole in the combined interpolated image 640, which isobtained by combining the images with a high ratio of the portionshaving no blurs. The combined interpolated image 640 is interpolated inthe original image. This allows an improvement of the image quality of amoving image whose frame rate is converted.

(Operation Example of Image Processing Apparatus)

FIG. 17 is a flowchart showing an example of an operation of the imageprocessing apparatus 200 in the first embodiment. This operation isstarted when the image processing apparatus 200 executes the processingto convert the frame rate, for example.

The image processing apparatus 200 determines whether an original imageis input or not (Step S910). When an original image is input (Step S910:Yes), the image processing apparatus 200 executes interpolated imagegeneration processing to generate a combined interpolated image (StepS920). Subsequently, the image processing apparatus 200 outputs theinterpolated image and the original image in a time-series order (StepS930). When an original image is not input (Step S910: No), or after theprocessing of Step S930 is performed, the image processing apparatus 200returns to the processing of Step S910.

FIG. 18 is a flowchart showing an example of the interpolated imagegeneration processing in the first embodiment. The interpolated imagegeneration unit 300 of the image processing apparatus 200 detects alocal motion vector for each block (Step S921). Subsequently, theinterpolated image generation unit 300 generates a normal interpolatedimage from original images based on the local motion vector (Step S922).

The interpolated image generation unit 300 extracts a high-frequencyarea from the original images (Step S923). Subsequently, theinterpolated image generation unit 300 detects a motion vector of thehigh-frequency area as a global motion vector (Step S924). Theinterpolated image generation unit 300 generates a high-frequency areainterpolated image from the original images based on the global motionvector (Step S925).

The interpolated image generation unit 300 determines a combinationratio for each pixel based on the high-frequency areas (Step S926).Subsequently, the interpolated image generation unit 300 combines thenormal interpolated image and the high-frequency area interpolated imageaccording to the combination ratio (Step S927). After Step S927, theinterpolated image generation unit 300 terminates the interpolated imagegeneration processing.

As described above, according to the first embodiment, the imageprocessing apparatus 200 combines the high-frequency area interpolatedimage and the normal interpolated image that are generated based on thechange in position of the high-frequency area, to generate aninterpolated image with no blurs in the high-frequency area.Additionally, the image processing apparatus 200 combines the normalinterpolated image and the high-frequency area interpolated image, togenerate an interpolated image in which a boundary between thehigh-frequency area and another area looks natural. Such an interpolatedimage is interpolated between the original images. This allows animprovement of the image quality of a moving image whose frame rate isconverted.

Modified Example

Although the local motion vector is detected to generate the normalinterpolated image in the first embodiment, the image processingapparatus 200 can also generate the normal interpolated image withoutdetecting the local motion vector. The image processing apparatus 200 ina modified example is different from that of the first embodiment inthat the local motion vector is not detected.

(Configuration Example of Interpolated Image Generation Unit)

FIG. 19 is a block diagram showing a configuration example of theinterpolated image generation unit 300 in the modified example of thefirst embodiment. The interpolated image generation unit 300 in themodified example is different from that of the first embodiment in thata normal interpolated image generation unit 325 is provided instead ofthe local motion vector detection unit 310 and the normal interpolatedimage generation unit 320.

The normal interpolated image generation unit 325 combines the originalimages F_(n) and F_(n-1) to generate the normal interpolated image MC_N.

FIG. 20 is a block diagram showing a configuration example of the normalinterpolated image generation unit 325 in the modified example of thefirst embodiment. The normal interpolated image generation unit 325includes an adder 323 and a multiplier 324.

The adder 323 adds the input values. The adder 323 receives an input ofa pixel value in the original image F_(n) and an input of a pixel valuein the original image F_(n-1). The adder 323 adds those input values andsupplies the resultant value to the multiplier 324.

The multiplier 324 multiplies the input value by a predetermined value.The multiplier 324 receives an input of the additional value suppliedfrom the adder 323. The multiplier 324 multiplies the input value by 0.5and supplies the resultant value, which is the pixel value “out”, to thecombination unit 370. In such a manner, the normal interpolated imagegeneration unit 325 generates a normal interpolated image withoutexecuting the motion compensation processing.

Note that the normal interpolated image generation unit 325 combines theoriginal images F_(n) and F_(n-1) to generate the normal interpolatedimage, but it may use one of the original images without change as thenormal interpolated image.

FIG. 21 is a diagram showing an example of a normal interpolated image660 in the modified example of the first embodiment. The normalinterpolated image 660 is an image generated without performing a motioncompensation, and hence blurs occur in a portion of a character telop661 that moves in the horizontal direction. The normal interpolatedimage 660 is combined with a high-frequency area interpolated imagehaving no blurs in the character telop. This suppresses the occurrenceof blurs in a combined interpolated image that is to be eventuallygenerated.

2. Second Embodiment

In the first embodiment, the image processing apparatus 200 amplifiesthe pixel value of the pixel in the high-frequency area according to thedifference. However, the difference may include a noise component and itis desirable to execute coring processing to remove the noise componentbefore the amplification. The image processing apparatus 200 of thesecond embodiment is different from that of the first embodiment in thatthe coring processing is performed on the difference.

(Configuration Example of Interpolated Image Generation Unit)

FIG. 22 is a block diagram showing a configuration example of theinterpolated image generation unit 300 in the second embodiment. Theinterpolated image generation unit 300 in the second embodiment isdifferent from that of the first embodiment in that a high-frequencyarea extraction unit 380 is provided instead of the high-frequency areaextraction unit 330.

The high-frequency area extraction unit 380 executes the coringprocessing to remove a value smaller than a difference threshold Th_(k1)from the difference values k, to generate a difference value k1.Additionally, the high-frequency area extraction unit 380 executes thecoring processing to remove a value smaller than a difference thresholdTh_(k2) from the difference values k, to generate a difference value k2.The high-frequency area extraction unit 380 amplifies the pixel valuesof the high-frequency areas HP_(n) and HP_(n-1) in accordance with thedifference value k1, to generate high-frequency areas HPK1_(n) andHPK1_(n-1). Additionally, the high-frequency area extraction unit 380amplifies the pixel values of the high-frequency areas HP_(n) andHP_(n-1) in accordance with the difference value k2, to generatehigh-frequency areas HPK2_(n) and HPK2_(n-1).

The high-frequency area extraction unit 380 supplies the high-frequencyareas HPK1_(n) and HPK1_(n-1) to the global motion vector detection unit340 and supplies the high-frequency areas HPK2_(n) and HPK2_(n-1) to thecombination ratio determination unit 360.

(Configuration Example of High-Frequency Area Extraction Unit)

FIG. 23 is a block diagram showing a configuration example of thehigh-frequency area extraction unit 380 in the second embodiment. Thehigh-frequency area extraction unit 380 is different from thehigh-frequency area extraction unit 330 in that coring processing units381 and 382, a multiplier 383, and an image memory 384 are additionallyprovided.

The coring processing unit 381 executes the coring processing to removethe difference value k smaller than the difference threshold Th_(k1).For example, the coring processing unit 381 uses the followingexpression to obtain the difference value k1 from the difference valuesk in the difference data K. The coring processing unit 381 suppliesdifference data K1 formed of the difference value k1 to the multiplier333. Note that the coring processing unit 381 is an example of a firstdifference coring processing unit in the section “What is claimed is”.

$\quad\begin{matrix}\{ \begin{matrix}{{k\; 1} = 0} & ( {k < {Th}_{k\; 1}} ) \\{{k\; 1} = k} & ( {k \geqq {Th}_{k\; 1}} )\end{matrix}  & {{Expression}\mspace{14mu} 20}\end{matrix}$

The multiplier 333 multiplies the pixel value in the high-frequency areaHP_(n) and the difference value k1 in the difference data K1. Data ofthe multiplication result, which serves as the high-frequency areaHPK1_(n), is supplied to the global motion vector detection unit 340.

The coring processing unit 382 executes the coring processing to removethe difference value k smaller than the difference threshold Th_(k2).For example, the coring processing unit 382 uses the followingexpression to obtain the difference value k2 from the difference valuesk in the difference data K. The coring processing unit 382 suppliesdifference data K2 formed of the difference value k2 to the multiplier383. Note that the coring processing unit 382 is an example of a seconddifference coring processing unit in the section “What is claimed is”.

$\begin{matrix}\{ \begin{matrix}{{k\; 2} = 0} & ( {k < {Th}_{k\; 2}} ) \\{{k\; 2} = k} & ( {k \geqq {Th}_{k\; 2}} )\end{matrix}  & {{Expression}\mspace{14mu} 21}\end{matrix}$

The multiplier 383 multiplies the input values. The multiplier 383receives an input of the pixel value in the high-frequency area HP_(n)and an input of the difference value k2 in the difference data K2. Themultiplier 383 multiplies those input values, supplies data of themultiplication result, which serves as the high-frequency area HPK2_(n),to the combination ratio determination unit 360, and causes the imagememory 384 to store the data.

Note that the high-frequency area extraction unit 380 executes both ofthe coring processing using the difference threshold Th_(k1) and thecoring processing using the difference threshold Th_(k2), but it mayexecute any one of the processing.

As described above, according to the second embodiment, the imageprocessing apparatus 200 executes the coring processing to remove thedifference smaller than the difference threshold, to remove the noisecomponent from the difference. This allows the global motion vector tobe easily detected and an appropriate combination ratio to be obtained.

Modified Example

In the second embodiment, the image processing apparatus 200 performsthe coring processing on the difference, but it may perform the coringprocessing on the high-frequency area. The image processing apparatus200 in the modified example is different from that of the secondembodiment in that the coring processing is performed on thehigh-frequency area.

(Configuration Example of High-Frequency Area Extraction Unit)

FIG. 24 is a block diagram showing a configuration example of thehigh-frequency area extraction unit 380 in the modified example of thesecond embodiment. The high-frequency area extraction unit 380 in themodified example is different from that of the second embodiment in thatcoring processing units 385 and 386 are provided instead of the coringprocessing units 381 and 382.

The coring processing unit 385 executes coring processing to remove apixel having a pixel value smaller than a pixel threshold Th_(p1) in thehigh-frequency area HP_(n). The coring processing unit 385 sequentiallysupplies the pixel value in the high-frequency area HP_(n), which isobtained after the coring processing, to the multiplier 333. Themultiplier 333 multiplies the difference value k and the pixel valuesupplied from the coring processing unit 385.

The coring processing unit 386 executes coring processing to remove apixel having a pixel value smaller than a pixel threshold Th_(p2) in thehigh-frequency area HP_(n). The coring processing unit 386 sequentiallysupplies the pixel value in the high-frequency area HP_(n), which isobtained after the coring processing, to the multiplier 383. Themultiplier 383 multiplies the difference value k and the pixel valuesupplied from the coring processing unit 386.

Note that the high-frequency area extraction unit 380 executes both ofthe coring processing using the pixel threshold Th_(p1) and the coringprocessing using the pixel threshold Th_(p2), but it may execute any oneof the processing. Further, the coring processing unit 385 is an exampleof a first pixel coring processing unit in the section “What is claimedis”. Furthermore, the coring processing unit 386 is an example of asecond pixel coring processing unit in the section “What is claimed is”.

As described above, according to the modified example, the imageprocessing apparatus 200 performs the coring processing on thehigh-frequency area HP_(n) to remove a noise component in thehigh-frequency area HP_(n).

Note that the embodiments described above are merely examples forembodying the present disclosure, and the matters in the embodimentshave correlations with the matters specifying the present disclosure inthe section “What is claimed is”. Similarly, the matters specifying thepresent disclosure in the section “What is claimed is” have correlationswith the matters in the embodiments of the present disclosure, which aredenoted by the same names as those in the section “What is claimed is”.Note that the present disclosure is not limited to the embodiments andcan be embodied by variously modifying the embodiments without departingfrom the gist of the present disclosure.

Additionally, the processing procedures described in the aboveembodiments may be considered as a method including a series of thoseprocedures, or may be considered as a program causing a computer toexecute the series of those procedures or as a recording medium storingthe program. Examples of the recording medium include a CD (CompactDisc), an MD (MiniDisc), a DVD (Digital Versatile Disk), a memory card,and a blu-ray disc (Blu-ray Disc (registered trademark)).

Note that the present disclosure can also take the followingconfigurations.

(1) An image processing apparatus, including:

a normal interpolated image generation unit configured to generate animage that is interpolated between a plurality of original imagesreproduced along time series, the image being a normal interpolatedimage, based on each of the plurality of original images;

a high-frequency area extraction unit configured to extract ahigh-frequency area having a spatial frequency higher than apredetermined value in each of the plurality of original images;

a high-frequency area interpolated image generation unit configured togenerate an image that is interpolated between the plurality of originalimages, the image being a high-frequency area interpolated image, basedon a change in position of the high-frequency area along with an elapseof time on the time series and on each of the plurality of originalimages; and

a combination unit configured to execute combining processing to combinethe normal interpolated image and the high-frequency area interpolatedimage.

(2) The image processing apparatus according to (1), further including avector detection unit configured to detect a vector indicating adirection and a distance in which a position of the high-frequency areachanges within a certain period of time on the time series, in which

the high-frequency area interpolated image generation unit is configuredto generate the high-frequency area interpolated image based on thevector and each of the plurality of original images.

(3) The image processing apparatus according to (2), further including acombination ratio determination unit configured to determine a ratio atwhich the normal interpolated image and the high-frequency areainterpolated image are combined, for each pixel in accordance with eachpixel value of pixels in the high-frequency area, in which

the combination unit is configured to execute the combining processingaccording to the ratio.

(4) The image processing apparatus according to (3), in which

the high-frequency area extraction unit includes

-   -   a filter unit configured to extract the high-frequency area in        each of the plurality of original images,    -   a difference detection unit configured to detect, for each        pixel, a difference in pixel value between two adjacent original        images of the plurality of original images, and    -   an increase unit configured to        -   increase each pixel value of the pixels in the            high-frequency area in each of the two adjacent original            images in accordance with the difference, and        -   supply the increased pixel value to the vector detection            unit and the combination ratio determination unit.            (5) The image processing apparatus according to (4), in            which

the high-frequency area extraction unit includes a first differencecoring processing unit configured to execute first difference coringprocessing to remove a difference smaller than a first differencethreshold from the detected differences, and

the increase unit is configured to

-   -   increase each pixel value of the pixels in the high-frequency        area in accordance with the difference on which the first        difference coring processing is performed, and    -   supply the increased pixel value to the vector detection unit.        (6) The image processing apparatus according to (4) or (5), in        which

the high-frequency area extraction unit includes a second differencecoring processing unit configured to execute second difference coringprocessing to remove a difference smaller than a second differencethreshold from the detected differences, and

the increase unit is configured to

-   -   increase each pixel value of the pixels in the high-frequency        area in accordance with the difference on which the second        difference coring processing is performed, and    -   supply the increased pixel value to the combination ratio        determination unit.        (7) The image processing apparatus according to any one of (4)        to (6), in which

the high-frequency area extraction unit includes a first pixel coringprocessing unit configured to execute first pixel coring processing toremove a pixel with a pixel value smaller than a first pixel valuethreshold in the extracted high-frequency area, and

the increase unit is configured to

-   -   increase, in accordance with the difference, each pixel value of        the pixels in the high-frequency area on which the first pixel        coring processing is performed, and    -   supply the increased pixel value to the vector detection unit.        (8) The image processing apparatus according to any one of (4)        to (7), in which

the high-frequency area extraction unit includes a second pixel coringprocessing unit configured to execute second pixel coring processing toremove a pixel with a pixel value smaller than a second pixel valuethreshold in the extracted high-frequency area, and

the increase unit is configured to

-   -   increase, in accordance with the difference, each pixel value of        the pixels in the high-frequency area on which the second pixel        coring processing is performed, and    -   supply the increased pixel value to the combination ratio        determination unit.        (9) The image processing apparatus according to any one of (1)        to (8), in which

each of the plurality of original images includes a plurality of blockseach having a predetermined shape, and

the normal interpolated image generation unit is configured to generatethe normal interpolated image based on a change in position of each ofthe plurality of blocks along with an elapse of time on the time seriesand on each of the plurality of original images.

(10) The image processing apparatus according to any one of (1) to (9),further including a selection unit configured to select the combinedimage by the combination unit and the plurality of original images inorder of the time series and output the selected image.(11) An image processing method, including:

generating, by a normal interpolated image generation unit, an imagethat is interpolated between a plurality of original images reproducedalong time series, the image being a normal interpolated image, based oneach of the plurality of original images;

extracting, by a high-frequency area extraction unit, a high-frequencyarea having a spatial frequency higher than a predetermined value ineach of the plurality of original images;

generating, by a high-frequency area interpolated image generation unit,an image that is interpolated between the plurality of original images,the image being a high-frequency area interpolated image, based on achange in position of the high-frequency area along with an elapse oftime on the time series and on each of the plurality of original images;and

executing, by a combination unit, combining processing to combine thenormal interpolated image and the high-frequency area interpolatedimage.

(12) A program causing a computer to execute:

generating, by a normal interpolated image generation unit, an imagethat is interpolated between a plurality of original images reproducedalong time series, the image being a normal interpolated image, based oneach of the plurality of original images;

extracting, by a high-frequency area extraction unit, a high-frequencyarea having a spatial frequency higher than a predetermined value ineach of the plurality of original images;

generating, by a high-frequency area interpolated image generation unit,an image that is interpolated between the plurality of original images,the image being a high-frequency area interpolated image, based on achange in position of the high-frequency area along with an elapse oftime on the time series and on each of the plurality of original images;and

executing, by a combination unit, combining processing to combine thenormal interpolated image and the high-frequency area interpolatedimage.

What is claimed is:
 1. An image processing apparatus, comprising: anormal interpolated image generation unit configured to generate animage that is interpolated between a plurality of original imagesreproduced along time series, the image being a normal interpolatedimage, based on each of the plurality of original images; ahigh-frequency area extraction unit configured to extract ahigh-frequency area having a spatial frequency higher than apredetermined value in each of the plurality of original images; ahigh-frequency area interpolated image generation unit configured togenerate an image that is interpolated between the plurality of originalimages, the image being a high-frequency area interpolated image, basedon a change in position of the high-frequency area along with an elapseof time on the time series and on each of the plurality of originalimages; and a combination unit configured to execute combiningprocessing to combine the normal interpolated image and thehigh-frequency area interpolated image.
 2. The image processingapparatus according to claim 1, further comprising a vector detectionunit configured to detect a vector indicating a direction and a distancein which a position of the high-frequency area changes within a certainperiod of time on the time series, wherein the high-frequency areainterpolated image generation unit is configured to generate thehigh-frequency area interpolated image based on the vector and each ofthe plurality of original images.
 3. The image processing apparatusaccording to claim 2, further comprising a combination ratiodetermination unit configured to determine a ratio at which the normalinterpolated image and the high-frequency area interpolated image arecombined, for each pixel in accordance with each pixel value of pixelsin the high-frequency area, wherein the combination unit is configuredto execute the combining processing according to the ratio.
 4. The imageprocessing apparatus according to claim 3, wherein the high-frequencyarea extraction unit includes a filter unit configured to extract thehigh-frequency area in each of the plurality of original images, adifference detection unit configured to detect, for each pixel, adifference in pixel value between two adjacent original images of theplurality of original images, and an increase unit configured toincrease each pixel value of the pixels in the high-frequency area ineach of the two adjacent original images in accordance with thedifference, and supply the increased pixel value to the vector detectionunit and the combination ratio determination unit.
 5. The imageprocessing apparatus according to claim 4, wherein the high-frequencyarea extraction unit includes a first difference coring processing unitconfigured to execute first difference coring processing to remove adifference smaller than a first difference threshold from the detecteddifferences, and the increase unit is configured to increase each pixelvalue of the pixels in the high-frequency area in accordance with thedifference on which the first difference coring processing is performed,and supply the increased pixel value to the vector detection unit. 6.The image processing apparatus according to claim 4, wherein thehigh-frequency area extraction unit includes a second difference coringprocessing unit configured to execute second difference coringprocessing to remove a difference smaller than a second differencethreshold from the detected differences, and the increase unit isconfigured to increase each pixel value of the pixels in thehigh-frequency area in accordance with the difference on which thesecond difference coring processing is performed, and supply theincreased pixel value to the combination ratio determination unit. 7.The image processing apparatus according to claim 4, wherein thehigh-frequency area extraction unit includes a first pixel coringprocessing unit configured to execute first pixel coring processing toremove a pixel with a pixel value smaller than a first pixel valuethreshold in the extracted high-frequency area, and the increase unit isconfigured to increase, in accordance with the difference, each pixelvalue of the pixels in the high-frequency area on which the first pixelcoring processing is performed, and supply the increased pixel value tothe vector detection unit.
 8. The image processing apparatus accordingto claim 4, wherein the high-frequency area extraction unit includes asecond pixel coring processing unit configured to execute second pixelcoring processing to remove a pixel with a pixel value smaller than asecond pixel value threshold in the extracted high-frequency area, andthe increase unit is configured to increase, in accordance with thedifference, each pixel value of the pixels in the high-frequency area onwhich the second pixel coring processing is performed, and supply theincreased pixel value to the combination ratio determination unit. 9.The image processing apparatus according to claim 1, wherein each of theplurality of original images includes a plurality of blocks each havinga predetermined shape, and the normal interpolated image generation unitis configured to generate the normal interpolated image based on achange in position of each of the plurality of blocks along with anelapse of time on the time series and on each of the plurality oforiginal images.
 10. The image processing apparatus according to claim1, further comprising a selection unit configured to select the combinedimage by the combination unit and the plurality of original images inorder of the time series and output the selected image.
 11. An imageprocessing method, comprising: generating, by a normal interpolatedimage generation unit, an image that is interpolated between a pluralityof original images reproduced along time series, the image being anormal interpolated image, based on each of the plurality of originalimages; extracting, by a high-frequency area extraction unit, ahigh-frequency area having a spatial frequency higher than apredetermined value in each of the plurality of original images;generating, by a high-frequency area interpolated image generation unit,an image that is interpolated between the plurality of original images,the image being a high-frequency area interpolated image, based on achange in position of the high-frequency area along with an elapse oftime on the time series and on each of the plurality of original images;and executing, by a combination unit, combining processing to combinethe normal interpolated image and the high-frequency area interpolatedimage.
 12. A program causing a computer to execute: generating, by anormal interpolated image generation unit, an image that is interpolatedbetween a plurality of original images reproduced along time series, theimage being a normal interpolated image, based on each of the pluralityof original images; extracting, by a high-frequency area extractionunit, a high-frequency area having a spatial frequency higher than apredetermined value in each of the plurality of original images;generating, by a high-frequency area interpolated image generation unit,an image that is interpolated between the plurality of original images,the image being a high-frequency area interpolated image, based on achange in position of the high-frequency area along with an elapse oftime on the time series and on each of the plurality of original images;and executing, by a combination unit, combining processing to combinethe normal interpolated image and the high-frequency area interpolatedimage.